% CS631 Advanced Programming in the UNIX Environment
% Author: Jan Schaumann <jschauma@netmeister.org>
\special{! TeXDict begin /landplus90{true}store end }

\documentclass[xga]{xdvislides}
\usepackage[landscape]{geometry}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{colordvi}

\newcommand{\smallish}{\fontsize{18}{18}\selectfont}

\begin{document}
\setfontphv

%%% Headers and footers
\lhead{\slidetitle}
\chead{CS631 - Advanced Programming in the UNIX Environment}
\rhead{Slide \thepage}
\lfoot{\Gray{Lecture 10: Things That Will Make Your Life Significantly Easier}}
\cfoot{\relax}
\rfoot{\Gray{\today}}

\vspace*{\fill}
\begin{center}
	\Hugesize
		CS631 - Advanced Programming in the UNIX Environment\\
		-- \\
		UNIX development tools \\
	\hspace*{5mm}\blueline\\ [1em]
	\Normalsize
		Department of Computer Science\\
		Stevens Institute of Technology\\
		Jan Schaumann\\
		\verb+jschauma@stevens.edu+\\
		\verb+https://stevens.netmeister.org/631/+
\end{center}
\vspace*{\fill}

\subsection{Software Development Tools}
\begin{center}
	\includegraphics[scale=0.6,angle=-90]{pics/ide.eps}
\end{center}

\subsection{Software Development Tools}
\begin{center}
	\includegraphics[scale=0.58]{pics/terminal.eps}
\end{center}

\subsection{Software Development Tools}
UNIX Userland is an IDE -- essential tools that follow the paradigm of ``Do
one thing, and do it right'' can be combined. \\

The most important tools are:
\begin{itemize}
	\item \verb+$EDITOR+
	\item the compiler toolchain
	\item {\tt gdb(1)} -- debugging your code
	\item {\tt make(1)} -- project build management, maintain program
		dependencies
	\item {\tt diff(1)} and {\tt patch(1)} -- report and apply differences
		between files
	\item {\tt cvs(1)}, {\tt svn(1)}, {\tt git(1)} etc. -- distributed project management,
		 version control
\end{itemize}

\subsection{EDITOR}
Know your \verb+$EDITOR+.  Core functionality:
\begin{itemize}
\item syntax highlighting
\item efficient keyboard maneuvering
\item setting markers, using buffers
\item copy, yank, fold e.g. blocks
\item search and replace
\item window splitting
\item autocompletion
\item jump to definition / manual page
\item applying external commands and filters
\end{itemize}

\subsection{EDITOR}
Examples given using {\tt vim(1)}. \\

Efficient keyboard maneuvering:

\begin{itemize}
	\item h, j, k, l
	\item w, b, e
	\item /, ?, \verb+^+, \verb+$+
	\item \verb+^+D, \verb+^+B
	\item zz, zt, zb
	\item \verb+%+, \verb+]}+, \verb+[{+
	\item :n, :prev, :rew
\end{itemize}

\subsection{EDITOR}
Examples given using {\tt vim(1)}. \\

Copy, yank, fold, markers, buffers etc.:
\begin{itemize}
	\item m \verb+[+a-zA-Z\verb+]+
	\item :marks
	\item v, V
	\item =
	\item d, y, "xy, "xp
	\item zf, zA
\end{itemize}

\subsection{EDITOR}
Examples given using {\tt vim(1)}. \\

Look-ups:
\begin{itemize}
	\item \verb+find /usr/src -name '*[ch]' -print | exec ctags -f ~/.ctgs+
	\item \verb|echo "set tags+=~/.ctags" >> ~/.vimrc|
	\item Ctrl+], Ctrl+t -- jump to definition and back
	\item K -- jump to manual page
	\item Ctrl+N -- autocomplete
\end{itemize}

\subsection{EDITOR}
Examples given using {\tt vim(1)}. \\

Integration with compiler, debugger, \verb+make(1)+
etc.

\begin{verbatim}
vim welcome.c
:make
Ctrl+]
:cnext
...
\end{verbatim}
\vspace{.5in}

Finally, two of your most powerful Unix IDE
integrations are a terminal multiplexer (e.g.
\verb+screen(1)+ or \verb+tmux(1)+) and copious use
of Ctrl+Z (i.e., the shell's job control mechanisms).

\subsection{EDITOR}
Examples given using {\tt vim(1)}. \\

\begin{center}
	\includegraphics[scale=0.4]{pics/vim.eps} \\
	\verb|https://duckduckgo.com/?q=vim+tutorial|
\end{center}



%\subsection{Compilers}
%
%\begin{center}
%	\includegraphics[scale=0.9,angle=-90]{pics/compiler_monster.eps}
%\end{center}
%

\subsection{Compilers}

A compiler translates {\em source code} from a high-level programming
language into {\em machine code} for a given architecture by performing a
number of steps:

\begin{itemize}
	\item lexical analysis
	\item preprocessing
	\item parsing
	\item semantic analysis
	\item code optimization
	\item code generation
	\item assembly
	\item linking
\end{itemize}

\subsection{Compilers}

\begin{center}
	\includegraphics[scale=0.65]{pics/compiler.eps}
\end{center}


\subsection{Compilers}

There are many different closed- and open-source compiler chains:

\begin{itemize}
	\item Intel C/C++ Compiler (or \verb+icc+)
	\item Turbo C / Turbo C++ / C++Builder (Borland)
	\item Microsoft Visual C++
	\item ...
\\

	\item Clang (a frontend to LLVM)
	\item GNU Compiler Collection (or \verb+gcc+)
	\item Portable C Compiler (or \verb+pcc+)
	\item ...
\end{itemize}

\subsection{The compiler toolchain}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\subsection{Preprocessing}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\begin{verbatim}
$ cd compilechain
$ cat hello.c
$ man cpp
$ cpp hello.c hello.i
$ file hello.i
$ man cc
$ cc -v -E hello.c > hello.i
$ more hello.i
$ cc -v -DFOOD=\"Avocado\" -E hello.c > hello.i.2
$ diff -bu hello.i hello.i.2
\end{verbatim}

\subsection{Compilation}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\begin{verbatim}
$ more hello.i
$ cc -v -S hello.i
$ file hello.s
$ more hello.s
\end{verbatim}

\subsection{Assembly}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\begin{verbatim}
$ as -o hello.o hello.s
$ file hello.o
$ cc -v -c hello.s
$ objdump -d hello.o
[...]
\end{verbatim}

\subsection{Linking}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\begin{verbatim}
$ ld hello.o
[...]
$ ld hello.o -lc
[...]
$ cc -v hello.o
[...]
$ ld -dynamic-linker /usr/libexec/ld.elf_so                 \
        /usr/lib/crt0.o /usr/lib/crti.o /usr/lib/crtbegin.o \
        hello.o -lc /usr/lib/crtend.o /usr/lib/crtn.o
$ file a.out
$ ./a.out
\end{verbatim}

\subsection{Linking}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).

\begin{verbatim}
$ cc -v -DFOOD=\"Avocado\" hello.c 2>&1 | more
\end{verbatim}


\subsection{{\tt cc(1)} and {\tt ld(1)}}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).
\\

Different flags can be passed to {\tt cc(1)} to be passed through to each tool
as well as to affect all tools.  \\

\begin{verbatim}
$ cc -v -O2 -g hello.c 2>&1 | more
\end{verbatim}

\subsection{{\tt cc(1)} and {\tt ld(1)}}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).
\\

Different flags can be passed to {\tt cc(1)} to be passed through to each tool
as well as to affect all tools.  \\

The order of the command line flags {\em may} play a role!
Directories searched for libraries via {\tt -L} and the resolving of undefined
symbols via {\tt -l} are examples of position sensitive flags.
\\

\begin{verbatim}
$ cc -v main.c -L./lib2 -L./lib -lldtest 2>&1 | more


$ cc -v main.c -L./lib -L./lib2 -lldtest 2>&1 | more
\end{verbatim}


\subsection{{\tt cc(1)} and {\tt ld(1)}}

The compiler chain or driver usually performs
preprocessing (e.g. via {\tt cpp(1)}), compilation
({\tt cc(1)}), assembly ({\tt as(1)}) and linking
({\tt ld(1)}).
\\

Different flags can be passed to {\tt cc(1)} to be passed through to each tool
as well as to affect all tools.  \\

The order of the command line flags {\em may} play a role!
Directories searched for libraries via {\tt -L} and the resolving of undefined
symbols via {\tt -l} are examples of position sensitive flags.
\\

The behavior of the compiler toolchain may be influenced by environment
variables (eg {\tt TMPDIR}, {\tt SGI\_ABI}) and/or the compilers default
configuration file (MIPSPro's {\tt /etc/compiler.defaults} or gcc's {\tt
specs}).

\begin{verbatim}
$ cc -v hello.c
$ TMPDIR=/var/tmp cc -v hello.c
$ cc -dumpspec
\end{verbatim}

\subsection{A Debugger}
\vspace*{\fill}
\begin{center}
	\includegraphics{pics/debugging.eps}
\end{center}
\vspace*{\fill}

\subsection{{\tt gdb(1)}}

The purpose of a debugger such as {\tt gdb(1)} is to allow you to see what is
going on ``inside'' another program while it executes -- or what another
program was doing at the moment it crashed. {\tt gdb} allows you to

\begin{itemize}
	\item make your program stop on specified conditions (for example by
		setting {\em breakpoints})
	\item examine what has happened, when your program has stopped (by looking
		at the {\em backtrace}, inspecting the value of certain variables)
	\item inspect control flow (for example by {\em stepping} through the
		program)
\end{itemize}
\vspace{.25in}
Other interesting things you can do:

\begin{itemize}
	\item examine stack frames: {\em info frame}, {\em info locals}, {\em info
		args}
	\item examine memory: {\em x}
	\item examine assembly: {\em disassemble func}
\end{itemize}


\subsection{{\tt gdb(1)}}
\begin{verbatim}
$ cd gdb-examples/ls
$ ./a.out
Memory fault (core dumped)
$ gdb ./a.out
(gdb) run ~/tmp

Program received signal SIGSEGV, Segmentation fault.
0x0000000000401a6d in record_stat
(statp=0x7f7fffffdaf0, path_name=0x7f7ff7b09175 "noown")
    at ls.c:497
497             strcpy(new_node->owner_name, password->pw_name);

(gdb) bt
(gdb) frame 0
(gdb) li
(gdb) print password
\end{verbatim}

\subsection{{\tt gdb(1)}}
\begin{verbatim}
(gdb) start -l
[...]
(gdb) watch blocksize
[...]
(gdb) c
[...]
(gdb) li
[...]
(gdb) show environment BLOCKSIZE
[...]
(gdb) call strtoll(blocksize_str, 0, 0)
[...]
\end{verbatim}

\subsection{{\tt make(1)}}

\begin{center}
	\includegraphics[scale=2.3]{pics/makerbot.eps}
\end{center}

\subsection{{\tt make(1)}}
{\tt make(1)} is a command generator and build utility. Using a
description file (usually {\em Makefile}) it creates a sequence of
commands for execution by the shell.

\begin{itemize}
	\item used to sort out dependency relations among files
\end{itemize}

\subsection{{\tt make(1)}}
{\tt make(1)} is a command generator and build utility. Using a
description file (usually {\em Makefile}) it creates a sequence of
commands for execution by the shell.

\begin{itemize}
	\item used to sort out dependency relations among files
	\item avoids having to rebuild the entire project after modification of a
		single source file
\end{itemize}

\subsection{{\tt make(1)}}
{\tt make(1)} is a command generator and build utility. Using a
description file (usually {\em Makefile}) it creates a sequence of
commands for execution by the shell.

\begin{itemize}
	\item used to sort out dependency relations among files
	\item avoids having to rebuild the entire project after modification of a
		single source file
	\item performs {\em selective} rebuilds following a {\em dependency graph}
\end{itemize}

\subsection{{\tt make(1)}}
{\tt make(1)} is a command generator and build utility. Using a
description file (usually {\em Makefile}) it creates a sequence of
commands for execution by the shell.

\begin{itemize}
	\item used to sort out dependency relations among files
	\item avoids having to rebuild the entire project after modification of a
		single source file
	\item performs {\em selective} rebuilds following a {\em dependency graph}
	\item allows simplification of rules through use of {\em macros} and {\em
		suffixes}, some of which are internally defined
\end{itemize}

\subsection{{\tt make(1)}}
{\tt make(1)} is a command generator and build utility. Using a
description file (usually {\em Makefile}) it creates a sequence of
commands for execution by the shell.

\begin{itemize}
	\item used to sort out dependency relations among files
	\item avoids having to rebuild the entire project after modification of a
		single source file
	\item performs {\em selective} rebuilds following a {\em dependency graph}
	\item allows simplification of rules through use of {\em macros} and {\em
		suffixes}, some of which are internally defined
	\item different versions of {\tt make(1)} (BSD make, GNU make, Sys V make,
		...) may differ (among other things) in
		\begin{itemize}
			\item variable assignment and expansion/substitution
			\item including other files
			\item flow control (for-loops, conditionals etc.)
		\end{itemize}
\end{itemize}

\subsection{{\tt make(1)}}

\begin{verbatim}
$ cd make-examples
$ ls *.[ch]
cmp.c       ls.c        main.c        stat_flags.c        util.c
extern.h    ls.h        print.c       stat_flags.h
\end{verbatim}

\begin{center}
	\includegraphics[scale=0.7,angle=-90]{pics/ls.eps}
\end{center}

\subsection{{\tt make(1)}}

\begin{verbatim}
$ cd make-examples
$ ls *.[ch]
cmp.c       ls.c        main.c        stat_flags.c        util.c
extern.h    ls.h        print.c       stat_flags.h
\end{verbatim}

\begin{center}
	\includegraphics[scale=0.7,angle=-90]{pics/ls-changed1.eps}
\end{center}

\subsection{{\tt make(1)}}

\begin{verbatim}
$ cd make-examples
$ ls *.[ch]
cmp.c       ls.c        main.c        stat_flags.c        util.c
extern.h    ls.h        print.c       stat_flags.h
\end{verbatim}

\begin{center}
	\includegraphics[scale=0.7,angle=-90]{pics/ls-changed2.eps}
\end{center}

\subsection{{\tt make(1)}}

\begin{verbatim}
$ cd make-examples
$ ls *.[ch]
cmp.c       ls.c        main.c        stat_flags.c        util.c
extern.h    ls.h        print.c       stat_flags.h
\end{verbatim}

\begin{center}
	\includegraphics[scale=0.7,angle=-90]{pics/ls-changed3.eps}
\end{center}

\subsection{{\tt make(1)}}

\begin{verbatim}
$ cd make-examples
$ ls *.[ch]
cmp.c       ls.c        main.c        stat_flags.c        util.c
extern.h    ls.h        print.c       stat_flags.h
\end{verbatim}

\begin{center}
	\includegraphics[scale=0.7,angle=-90]{pics/ls-changed4.eps}
\end{center}

\subsection{{\tt make(1)}}
\begin{verbatim}
$ ln -s Makefile.1 Makefile
$ make # or: make -f Makefile.1
[...]
$ make
\end{verbatim}

\subsection{{\tt make(1)}}
\begin{verbatim}
$ ln -sf Makefile.2 Makefile
$ make # or: make -f Makefile.2
[...]
$ make clean
$ export CFLAGS="-Wall -Werror"
$ make
[...]
$ make clean
[...]
\end{verbatim}

\subsection{{\tt make(1)}}
\begin{verbatim}
$ make -f Makefile.3
[...]
$ make -f Makefile.3 ls.txt
[...]
$
\end{verbatim}

\subsection{{\tt make(1)}}
\smallish
\begin{verbatim}
$ ln -sf Makefile.4 Makefile
$ make help
[...]
$ make showvars
[...]
$ make CFLAGS="${CFLAGS}" showvars
[...]
\end{verbatim}
\Normalsize

\subsection{Priority of Macro Assignments for {\tt make(1)}}

\begin{enumerate}
	\item Internal (default) definitions of {\tt make(1)}
	\item Current shell environment variables.  This includes macros that you
		enter on the {\em make} command line itself.
	\item Macro definitions in {\em Makefile}.
	\item Macros entered on the {\tt make(1)} command line, if they follow
		the {\em make} command itself.
\end{enumerate}

\subsection{Ed is the standard text editor.}
\begin{verbatim}
$ ed
?
help
?
quit
?
exit
?
bye
?
eat flaming death
?
^C
?
^D
?
\end{verbatim}

\subsection{Ed is the standard text editor.}
\begin{verbatim}
$ ed
a
ed is the standard Unix text editor.
This is line number two.
.
2i

.
%l
3s/two/three/
w foo
q
$ cat foo
\end{verbatim}

\subsection{{\tt diff(1)} and {\tt patch(1)}}
{\tt diff(1)}:
\begin{itemize}
	\item compares files line by line
	\item output may be used to automatically edit a file
	\item can produce human ``readable'' output as well as diff entire
		directory structures
	\item output called a {\em patch}
\end{itemize}

\subsection{{\tt diff(1)} and {\tt patch(1)}}
{\tt patch(1)}:
\begin{itemize}
	\item applies a {\tt diff(1)} file (aka {\em patch}) to an original
	\item may back up original file
	\item may guess correct format
	\item ignores leading or trailing ``garbage''
	\item allows for reversing the patch
	\item may even correct context line numbers
\end{itemize}

\subsection{{\tt diff(1)} and {\tt patch(1)}}
\begin{verbatim}
$ diff Makefile.2 Makefile.5
[...]
$ cp Makefile.2 /tmp
$ ( diff -e Makefile.2 Makefile.5; echo w; ) | ed Makefile.2
$ diff Makefile.[25]
$ mv /tmp/Makefile.2 .
$ diff -c Makefile.[25]
$ diff -u Makefile.[25] > /tmp/patch
$ patch </tmp/patch
$ diff Makefile.[25]
\end{verbatim}

\subsection{{\tt diff(1)} and {\tt patch(1)}}
Difference in {\tt ls(1)} between NetBSD and OpenBSD:
\begin{verbatim}
$ diff -bur netbsd/src/bin/ls openbsd/src/bin/ls
\end{verbatim}

\addvspace{.5in}
Difference in {\tt ls(1)} between NetBSD and FreeBSD:
\begin{verbatim}
$ diff -bur netbsd/src/bin/ls freebsd-ls/ls
\end{verbatim}

\subsection{Revision Control}
Version control systems allow you to

\begin{itemize}
	\item collaborate with others
	\item simultaneously work on a code base
	\item keep old versions of files
	\item keep a log of the who, when, what, and why of any changes
	\item perform release engineering by creating {\em branches}
\end{itemize}

\subsection{Revision Control}
\begin{itemize}
	\item Source Code Control System ({\em SSCS}) begat the Revision
		Control System ({\em RCS}).
	\item RCS operates on a single file; still in use for misc. OS
		config files
	\item the Concurrent Versions System ({\em CVS}) introduces a
		client-server architecture, control of hierarchies
	\item {\em Subversion} provides atomic commits, renaming, cheap
		branching etc.
	\item {\em Git}, {\em Mercurial} etc. implement a {\em
		distributed} approach (ie peer-to-peer versus
		client-server), adding other features (cryptographic
		authentication of history, ...)
\end{itemize}

\subsection{Revision Control}
Examples:

{\tt http://cvsweb.netbsd.org/bsdweb.cgi/src/bin/ls/} \\

{\tt http://svnweb.freebsd.org/base/stable/9/bin/ls/} \\

{\tt http://git.savannah.gnu.org/cgit/coreutils.git/log/} \\

{\tt http://cvsweb.netbsd.org/bsdweb.cgi/src/share/misc/bsd-family-tree?rev=HEAD}

% \subsection{Revision Control}
% Git example:
% \begin{verbatim}
% $ cd freebsd
% $ git diff >/tmp/diff
%
% # fork repository
% $ git remote add jschauma git@github.com:jschauma/freebsd.git
% $ git commit .
% $ git push jschauma master
%
% # pull request
% \end{verbatim}

\subsection{Revision Control: Branching}
Different strategies:
\begin{itemize}
	\item trunk / master is fragile
		\begin{itemize}
			\item {\em trunk} is work in progress, may not even compile
			\item all work happens in {\em trunk}
			\item releases are tagged on {\em trunk}, then branched
		\end{itemize}
	\item trunk / master is stable
		\begin{itemize}
			\item {\em master} is always stable
			\item all work is done in branches (feature or bugfix)
			\item feature branches are deleted after merge
			\item releases are made automatically from master
		\end{itemize}
\end{itemize}

You may combine these as {\em release branching} /
{\em feature branching} / {\em task branching}.

\subsection{Commit Messages}

Commit messages are like comments: too often useless
and misleading, but critical in understanding human
thinking behind the code. \\

Commit messages should be full sentences in correct
and properly formatted English. \\

Commit messages briefly summarize the {\em what}, but
provide important historical context as to the {\em
how} and, more importantly, {\em why}. \\

Commit messages SHOULD reference and integrate with
ticket tracking systems. \\

See also:
\begin{itemize}
	\item \verb+http://is.gd/Wd1LhA+
	\item \verb+http://is.gd/CUtwhA+
	\item \verb+http://is.gd/rPQj5E+
\end{itemize}

\subsection{Revision Control}
\begin{verbatim}
$ cd freebsd/bin/ls
$ git log | cat
\end{verbatim}

\subsection{Links}
Revision Control: \\
{\tt http://cvsbook.red-bean.com/cvsbook.html} \\

{\tt http://svnbook.red-bean.com/} \\

{\tt http://git-scm.com/} \\

GDB: \\
{\tt http://sources.redhat.com/gdb/current/onlinedocs/gdb\_toc.html} \\

\verb+http://heather.cs.ucdavis.edu/~matloff/UnixAndC/CLanguage/Debug.html+ \\

{\tt http://www.unknownroad.com/rtfm/gdbtut/gdbtoc.html}


\end{document}
